Orthogonal Matching Pursuit with Replacement

In this paper, we consider the problem of compressed sensing where the goal is to recover all sparsevectors using a small number of fixed linear measurements. For this problem, we propose a novelpartial hard-thresholding operator that leads to a general family of iterative algorithms. While oneextreme of the family yields well known hard thresholding algorithms like ITI and HTP[17, 10], theother end of the spectrum leads to a novel algorithm that we call Orthogonal Matching Pursuit withReplacement (OMPR). OMPR, like the classic greedy algorithm OMP, adds exactly one coordinateto the support at each iteration, based on the correlation with the current residual. However, unlikeOMP, OMPR also removes one coordinate from the support. This simple change allows us to provethat OMPR has the best known guarantees for sparse recovery in terms of the Restricted IsometryProperty (a condition on the measurement matrix). In contrast, OMP is known to have very weakperformance guarantees under RIP. Given its simple structure, we are able to extend OMPR usinglocality sensitive hashing to get OMPR-Hash, the first provably sub-linear (in dimensionality) al-gorithm for sparse recovery. Our proof techniques are novel and flexible enough to also permit thetightest known analysis of popular iterative algorithms such as CoSaMP and Subspace Pursuit. Weprovide experimental results on large problems providing recovery for vectors of size up to milliondimensions. We demonstrate that for large-scale problems our proposed methods are more robustand faster than existing methods.